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Abstract 



The metric structure of synaptic connections is obviously an impor- 
tant factor in shaping the properties of neural networks, in particular the 
capacity to retrieve memories, with which are endowed autoassociativc 
nets operating via attractor dynamics. Qualitatively, some real networks 
in the brain could be characterized as 'small worlds', in the sense that the 
structure of their connections is intermediate between the extremes of an 
orderly geometric arrangement and of a geometry-independent random 
mesh. Small worlds can be defined more precisely in terms of their mean 
path length and clustering coefficient; but is such a precise description 
useful to better understand how the type of connectivity affects memory 
retrieval? 

We have simulated an autoassociativc memory network of intcgrate- 
and-fire units, positioned on a ring, with the network connectivity varied 
parametrically between ordered and random. We find that the network 
retrieves when the connectivity is close to random, and displays the char- 
acteristic behavior of ordered nets (localized 'bumps' of activity) when 
the connectivity is close to ordered. Recent analytical work shows that 
these two behaviours can coexist in a network of simple threshold-linear 
units, leading to localized retrieval states. We find that they tend to be 
mutually exclusive behaviours, however, with our integrate-and-fire units. 
Moreover, the transition between the two occurs for values of the con- 
nectivity parameter which are not simply related to the notion of small 
worlds. 
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1 Introduction 



Autoassociative memory retrieval is often studied using neural network models 
in which connectivity does not follow any geometrical pattern, - i.e. it is either 
all-to-all or, if sparse, randomly assigned. Such networks can be characterized 
by their storage capacity, expressed as the maximum number of patterns that 
can be retrieved, which turns out to be proportional to the number k of con- 
nections per unit. Networks with a regular geometrical rule informing their 
connectivity, instead, can often display geometrically localized patterns of ac- 
tivity, i.e. stabilize into activity profile 'bumps' of width proportional to k. 
Recently, applications in various fields have used the fact that small-world net- 
works, characterized by a connectivity intermediate between regular and ran- 
dom, have different graph theoretic properties than either regular or random 
networks. In particular, small-worlds have both a high clustering coefficient C 
and a short characteristic path length L, at the same time |19| . 

Note that while autoassociative retrieval is an obvious model for a physio- 
logical memory process, also self-sustaining localized states are of interest for 
the real brain, as models of the short-term memory of a location on a cortical 
map. The two behaviours are combined in models of cortical function in which 
a localized and input-specific pattern of activity represents the memory of a 
given stimulus in a given position on the sensory array jl5| . 

Here we consider a family of ID networks of integrate-and-fire units, where 
changing the parameter of randomness q allows to go from a regular network 
{q = 0) in which all connections are relatively short-range, to a spatially random 
network (q = 1) in which all connections can be considered to be long-range. 
Intermediate values of q correspond to a biologically plausible combination of 
long- and short-range connections. For q = such a network spontaneously 
forms activity bumps ^ , and this property is expected to be preserved at least 
for small values of q. For q = 1 the network behaves as an ordinary autoasso- 
ciative net in in J and likewise this is expected to hold also for q < 1, as it does 
with a simpler binary-unit model jl2| . 

Can activity bumps and retrieval coexist? 

We first explore how both the ability to retrieve, and to form bumps, change 
with the degree of randomness q in the connectivity. Recent analytical work, 
on a model in which the connectivity has a Gaussian spread of variable width, 
in fact shows that retrieval and localization coexist below a critical width value 
(and a critical storage load) jJS]- We can then address the question of whether 
the two abilities can coexist also with our connectivity model, at least in an 
intermediate regime between g ~ and g ~ 1. 

Is the transition between the different regimes related to small worlds? 

Since the ability to form bumps depends on the mutual support among active 
units through excitatory feedback loops, one can expect it to persist as long as 
the clustering coefficient C remains large, even as q grows above 0. Conversely, 
since pattern retrieval depends on the propagation of the retrieval signal to 
all units in the network independently of their location, one can expect robust 
retrieval to require that the mean path length from one unit to any other one is 
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still short, even if q is below 1. Thus, one can expect an intermediate small- world 
regime of coexistence between bumps and retrieval. 

Can they coexist in more realistic networks, with integrate-and-fire dynam- 
ics? 

Bohland and Minai 0, and more recently McGraw and Menziger [T^ and 
Morelli et al ^21 have considered somewhat related issues with model networks 
of binary neurons, in which localization and retrieval do not appear to coex- 
ist (an observation explained by a simple argument proposed by Yasser Roudi 
(personal communication)). By contrast, Roudi and Treves find that they co- 
exist with threshold-linear units ^Sl- The essential difference between these 
two very simple and dynamically implausible models, for the stability of bump 
states and of retrieval states, is whether single-unit activity levels saturate or 
not. Integrate-and-fire units with an absolute refractory period saturate to an 
intermediate and variable degree (depending, effectively, on how far below sat- 
uration they normally operate [Tl \14\ ). They provide therefore a dynamically 
more plausible, but still very simple model to approach the issue of the coexis- 
tence of localization and retrieval in a cortical setting. This is what we discuss 
here, presenting the result of computer simulations. 



2 Model 

2.1 Integrate-and-Fire Model 

We consider a family of ID networks of N integrate-and-fire units jJTj , arranged 
on a ring. In the simplest possible integrate-and-fire model we adopt, the geom- 
etry of a neuron is reduced to a point, and the time evolution of the membrane 
potential Vi of cell i, i = l,...,iV during an interspike interval follows the 
equation 

mt}_ ^ _YM^mi + ?2IL^py^(t) + r^\t) + h{t)) (1) 

where Tm is the time constant of the cell membrane, Rm its passive resistance, 
and V"^^^ the resting potential. The total current entering cell i consists of a 
synaptic current /'^" (t) due to the spiking of all other cells in the network that 
connect to cell i, an inhibitory current /*"''(t), which depends on the network 
activity, and a small external current Ii{t), which represents all other inputs 
into cell i. 



The last, external component of the current can be expressed through a di- 
mensionless variable Xi{t) as 

h{t) = \,{t)(V"' -V^''')/R^ (2) 



We use a simplified model for the inhibition, where it depends on the excitatory 
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activity of the network. The inhibitory current is the same for all neurons at a 
given moment of time, and changes in time according to the equation 

dpg^ = -^r-\t) + y-^ ^^"-J^^'K {t) (3) 

at Tinh -/ V Km 

where A*"'' is a dimensionless inhibitory parameter of the model and r{t) is a 
rate function characterizing the network activity up to time t: 

= EET K{t-t')5{t'-U)dt' (4) 

i {t.} °° 

The kernel K{t~t') wc use for our simulations is a square wave function lasting 
one time step A< of the integration, so that r{t) is just a total number of spikes 
in the time interval At. 

As Vi reaches a threshold value y*'*'", the cell emits a spike, after which the 
membrane potential is reset to its resting value V'''^^ and held there for the 
duration of an absolute refractory period Tre} ■ During this time the cell cannot 
emit further spikes no matter how strongly stimulated. 

2.2 Synaptic Inputs 

All synapses between neurons in the network are excitatory, and the synaptic 
conductances are modeled as a difference of two exponents with time constants 
Ti and r2, such that T2 is of the order of Tm and t\ » T2. The synaptic current 
entering cell i at time t then depends on the history of spiking of all cells that 
have synapses on i: 

. g.^j:^(e-^_e-^) (5, 

JT^t {tj} 

where {tj} are all the times before time t when neuron j produced a spike and 
AVji is a size of the excitatory postsynaptic potential (EPSP) in neuron i evoked 
by one presynaptic spike of neuron j. 

The relative size of the evoked EPSP determines the efficacy of synapse ji: 



(V^h _ yres\ 



where A"^" is the model parameter characterizing synaptic excitation, Cji is 
or 1 depending on the absence or presence of the connection from neuron j to 
neuron i, and is modification of synaptic strength due to Hebbian learning, 
which will be described later (Section 2.4). 
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2.3 Connectivity 



In our model, all connections in the network are uni-directional, so the con- 
nectivity matrix is not symmetrical. We use a special procedure for creating 
connections, after |19| : the probability of one neuron to make a connection to 
another depends on a parameter q in such a way, that changing q from to 1 
allows one to go from a regular to a spatially random network. 

A regular network {q = 0) in this case is a network where connections are 
formed based on a Gaussian distribution. The probability to make a connection 
from neuron i to neuron j in such a metwork is 

Po(cy) = (7) 

where |« — j| is the minimal distance between neurons i and j on the ring. This 
results in an average number of connections per neuron k « •\/27rcr (actually, 
k < •\/27rcr because of the finite length of the ring) . 

In a random network (g — 1), the probability of creating a connection does 
not depend on the distance between neurons: 

Pi(c,,) - k/N (8) 

Intermediate values of q thus correspond to a biologically plausible combination 
of long- and short-range connections: 

P,{c,,) = {l-q)e-^ +q^ (9) 

The parameter q is referred to as the 'randomness' parameter, following Watts 
and Strogatz, who used a similar procedure for interpolating between regular 
and random graphs. 

As originally analyzed by Watts and Strogatz the average path length and 
clustering coefficient in a graph wired in this way both decrease with q, but 
their functional dependence takes different shapes. Thus, there is a range of q 
where the clustering coefficient still remains almost as high as it is in a regu- 
lar graph, while the average path length already approaches values as low as 
for random connectivity. Graphs that fall in this range are called small-world 
graphs, referring to the fact that for the first time this type of connectivity 
was explored in a context of social networks jjl]. In this context the average 
(or characteristic) path length L can be thought of as the average number of 
friends in the shortest chain connecting two people, and the clustering coeffi- 
cient C as the average fraction of one's friends who are also friends of each other. 

In a neural network, L corresponds to the average number of synapses that 
need to be passed in order to transmit a signal from one neuron to another, and 
thus depends on the number and distribution of long-range connections. The 
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clustering coefficient C characterises the density of local connections, and can 
be thought of as the probability of closing a trisynaptic loop (the probability 
that neuron k is connected to i, given that some neuron j which receives from 
i connects to k). 



2.4 Hebbian plasticty 

Associative memory in the brain can be thought of as mediated by Hebbian 
modifiable connections. Modifying connection strength corresponds to modify- 
ing the synaptic efficacy of Eq. El 




where 77^ is the firing rate of neuron i in the /ith memory pattern, {rj) is the 
average firing rate, the network stores p patterns with equal strength, and M is 
a normalization factor. The normalization factor should be chosen to maximally 
utilize the useful range of synaptic efficacy, which could be done by equalizing 
its variance with its mean. In order to preserve the non-negativity of each effi- 
cacy, in Eq. Elwe set = — 1 whenever (1 + J.y ) < 0. 

We draw patterns of sparseness a from a binary distribution, so that 77'' — l/a 
with probability a and 77^ = with probability 1 — a. The 'average firing rate' 
(77) in this case is equal to 1. 



2.5 Experimental Paradigm 

After the connections have been assigned and the synaptic efficacies modified 
by storing memory patterns, we randomize the starting membrane potentials 
for all the neurons: 

= V"" + /5(T/*'* - (11) 

where /3 = 0.1 is a constant and rii is a random number between and 1, gen- 
erated separately for each neuron i. We then let the network run its dynamics. 

After a time a cue for one of the p patterns (for instance, pattern jj) is 
given, in the form of an additional cue current injected into each cell. The cue 
current is injected at a constant value up to the time ti, and then decreased 
linearly to so that it becomes at time ^2- Thus, for the dimensionless current 
in (2) we have 

A.(0=A° + A-/(t)(7yf-l) (12) 
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where A" > is a constant, diniensionless external current flowing into all cells, 
A'^"'^ is a dimensionless cue current parameter, and 





fit) = { 1 



t2-tl 



t <to or t2 <t 

to<t<ti (13) 

ti<t<t2 



If the cue is partial - for example, the cue quality p = 25% - then a continuous 

patch of the network, comprised of 25% of the length along the network ring, 
receives the cue for pattern whereas the rest of the network receives a random 
pattern. (In the simulations, the random pattern on the rest of the network has 
a higher average activity than the original patterns, which makes the overall 
network activity rise slightly during application of the cue.) 

During the time tgim of the simiilation. spike counts (numbers of spikes fired by 
each neuron), are collected in sequential time windows of length twin- 



nit) = E /* ^i*' - *')'^*' 



(14) 



We then look at the geometrical shape of the network activity profile in each 
window, and at the amount of overlap of this activity with each of the patterns 
stored. 

The amount of overlap with pattern jj, in the time window ending at time t 
is calculated as the cosine of the angle between the pattern and the spike count 
vector: 

Oi^it) = T,iV^ri{t ) ^^^^ 



The measure of memory retrieval is 



^ \0'^{tsim) ^chanceit 



m = , 



chance K^sim ) 



^chanceitsim) 



(16) 



where O'^f^^^^^^t) is the chance level of the overlap when the cue for pattern n 
is given: 

O'cHanceit) = E ^ (1^) 

Note that the retrieval measure m may occasionally become negative. This 
occurs when retrieval is so poor that, even though a cue for pattern has been 
given, the network activity happens to have less overlap with fi than it has, on 
average, with the other stored patterns. 
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To quantify the prominence of the bump in the network activity profile, we 
compute the standard deviation of the position of each neuron weighted by its 
activity, with respect to the center of mass of the activity profile: 



Ei \i-icM\^ri{t) 



(18) 



The ratio of the standard deviation cro of a uniform distribution to (Ja{t) has an 
intuitive meaning as the number of bumps that can be fit on the ring next to 
each other. We take the measure of 'bumpiness' to be 



For an easier comparison with the memory retrieval measure, we compute the 

'bumpiness' of a particular network in the last time window of the simulation 
{t = tsim)i and then average over simulations with different cues. 

A rough measure of the relevance of saturation effects to the activity of sin- 
gle units is the ratio if) between the firing rate of the most active units in a 
localized state or in a retrieval state, and their maximum possible firing rate, 
given by the inverse of the absolute refractory period. In fact, for binary units 
= 1 and for threshold-linear units tjj = 0. In our simulations, the most active 
units typically fired at 120 Hz and the absolute refractory period was 3 msec, 
implying ?/) ~ 0.3. 

Table 1: Parameters Used for the Simulations. 



m = 



o-»(t) 



(19) 



f-1 
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Quantity 


Symbol 


Value 


Number of cells 


N 


1000 


Average number of outgoing connections per cell 


k 


24 - 208 


ParMiu(>t(^r of raiidoium^sK 


<J 


- 1 


Synaptic excitation constant 


\syn 


40 


Inhibition constant 




20 


External excitatory current parameter 


AO 


0.25 


Cue current parameter 




0.1 


Number of memory patterns stored 


p 


5 


Memory sparseness 


a 


0.2 


Synaptic plasticity normalization factor 


M 


10 


Cue quality 


P 


5 - 100% 


Membrane time contant 




5 msec 


Synaptic conductance time constant 1 


n 


30 msec 


Synaptic conductance time constant 2 


T2 


4 msec 


Inhibition time constant 




4 msec 


Absolute refractory period 


Tref 


3 msec 


Integration time step 


At 


0.1 m,sec 


Simulation time 


isim 


1000 msec 


Cue onset time 


to 


150 msec 


Time when the cue starts to decrease 


h 


300 msec 


Cue offset time 




500 msec 


Sampling time window 


twin 


50 msec 



Note: Ranges are indicated for quantities that varied within runs. 



3 Results 

3.1 Graph-Theoretic Properties of the Network 

Finding an analytical approximation for the characteristic path length L is not 
a straightforward calculation. Estimating the clustering cocfhcic^nt C for a given 
network is, however, quite easy. In our case, integrating connection probabilities 
(assuming, for the sake of simplicity, an infinite network) yields the analytical 
estimate 

(1-9)' + |f (20) 

or, after normalization, 

C'(g) = ^M^^ = (l-g)3 (21) 

which does not depend either on the network size N, or on the number k of 
connections per neuron. 



C{q) 



71" iv 
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Once connections have been assigned, both L and C can be calculated nu- 
merically. Keeping the number of connections per neuron A: > In ensures that 
the graph is connected [7j, i-e. that the number of synapses in the shortest path 
between any two neurons does not diverge to infinity. 

The results of numerical calculations are shown in Fig. The characteris- 
tic path length and the clustering coefficient both decrease monotonically with 
g, taking their maximal values in the regular network and minimal values in the 
random network. Similar to what was described by Watts and Strogatz|19 j |18 | . 
L and C decrease in different fashions, with L dropping down faster than C. 
For example, for q — 0.2, C" is at 52% of its maximal value C"(0) = 1, while 
L — Lmin bas already dropped to 8% of L(0) — Lmm- The inset shows that L 
approaches it maximal value L(0) only for q < 10"'*. 

Comparing the clustering coefficient estimated analytically with the simplified 
formula H2U|I and the one calculated numerically shows some minor discrepancy 
(Fig.^), which is due to the fact that analytical calculations were performed 
for an infinite network. 

In conclusion, over a broad range 0.001 < q < 0.2 one can characterize the 
mean path length as short, and the clustering coefficient as large, and one can 
therefore expect simulations with e.g. q — 0.05 or q — 0.1 to demonstrate char- 
acteristic small- world behaviour. 



3.2 Bumps Formation in a Regular Network 

Networks whose connectivity follows a strict geometrical order spontaneously 
form bumps in their activity profile, as shown in Fig. |21 Bumps are formed 
before the presentation of a pattern specific cue, and can either persist, or be 
temporarily or permanently altered or diplaced after cue onset. Robust bumps, 
which remain after the cue has been introduced and then removed, tend to form 
only for regular or nearly regular connectivity (small values of q). Fig.^^ ex- 
emplifies this behavior for a network with fc = 41 connections per unit, p = 0.25 
and different values of g. In the example, a robust single bump forms for q ^ 
and q = 0.2. A double bump forms for q = 0.4, and is relocated to a different 
section of the ring following the cue. A very noisy triple bump is barely notice- 
able towards the end of the simulation for q — 0.6, and no bumps can be really 
observed for random or nearly random nets, q — 0.8 and q — I. Thus bump 
formation appears to be strongly favored by a regular connectivity, as expected. 
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3.3 Memory Retrieval in a Random Network 

Fig. 03 shows the overlaps with the different memory patterns in the same 
simulations used for Fig. with k = 41. The cue which is presented after 
150msec is correlated with the second pattern along one fourth of the ring, and 
it reliably enhances the overlap with the second pattern during cue presenta- 
tion, whatever the value of q. By chance, the second (and to some extent the 
fifth) pattern tends to have an elevated initial overlap also before the presenta- 
tion of the cue, a behaviour often observed in autoassociative nets and due to 
fluctuations. Despite this early 'advantage', once the pattern specific cue has 
been completely removed (after 500msec, that is after the first 10 frames in the 
Figure) the overlap with the cued pattern returns to chance level, except in the 
case of a fully random (g = 1) or nearly random {q = 0.8) connectivity. Thus the 
randomness in the connectivity appears to favor memory retrieval, as expected, 
presumably by ensuring a more rapid propagation of the retrieval signal to the 
whole network, along shorter mean paths. 

3.4 Bumps and Memory in Networks of Different Degrees 
of Order 

Fig. ^ summarizes the results of simulations like those of Fig. |3| for fc = 41 
and a fully extended cue (p = 100%). The spontaneous activity bumps, which 
are formed in the regular network, can be observed, somewhat reduced, up to 
q « 0.4 0.5. For larger values of q the bumpiness measure quickly drops to 
a value close to zero. Storing random binary patterns on the network does not 
affect the bumps, but the retrieval performance appears to be very poor for 
small q, though the overlap measure with the pattern to be retrieved remains 
marginally above chance. As the randomness increases {q > 0.6), robust re- 
trieval appears even though the normalized overlap is far from its maximum at 
1. Still, for this value of the number of connections (fc = 41), and for full cues, 
the transitions from bumps to no bumps and from no retrieval to retrieval both 
appear to be fairly sharp, and both occur between q = 0.4 and q = 0.6. 

Results are somewhat clearer when considering partial cues, as shown in Fig.|31 
which reports the retrieval measure averaged over all 5 cued patterns and over 10 
network realizations, and as a function of cue quality. Despite the considerable 
variability (see the large standard deviations), the average retrieval performance 
appears to depend smoothly on q and p. For partial but still sufficiently large 
cues, 75% > p > 25%, the network retrieves reasonably well when it is regular 
or nearly regular, q > 0.7, approaching the performance obtained with full cues. 
The overlap is reduced for q = 0.6, and it approaches zero for q < 0.5. For 
p = 15%, retrieval appears for larger q and is weaker, and for even lower cue 
quality, p = 5% — 10%, essentially no retrieval occurs. 

The non-zero average overlap measures reached by regular networks with full 
cues presumably reflect the network occasionally freezing, i.e. getting stuck in 
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a particular firing configuration, what may be called remnant magnetization in 
a spin system. This makes the distinction between retrieval and no retrieval 
somewhat fuzzier. Nevertheless, whatever the size of the cue, it is apparent 
that there is a well defined transition between two behaviors, bump formation 
and retrieval, which for fc = 41 is concentrated between q = 0.5 and q = 0.6. 
The region of coexistence, if it exists, is in this case very limited. 

Changing k does not affect the qualitative network behavior, as shown in Fig.0 
for the case of full cues and in Fig. [7| in the case of partial cues {p = 50%). 
Whatever the connectivity k there is a transition between bump formation in 
nearly regular networks (on the left in each panel) and retrieval in nearly ran- 
dom networks (on the right). The transition is not always very sharp, and in 
some cases one could describe an individual network as able to both form bumps 
and retrieve memory patterns to some extent. When the number of connections 
is very limited (fc = 24), both retrieval and bump formation are weak, when 
they at all occur. For more numerous connections the degree of bumpiness, as 
quantified by our measure, reaches for regular networks values close to 0.6 with 
full cues, and up to 0.75 with partial cues. 

In Fig. 13 the range of the parameter of randomness, over which bump formation 
occurs, shrinks with increasing fc, and in the case of fc = 208 the bumpiness for 
q = 0.2 is already down to roughly 1/3 of its value for the regular network. The 
retrieval measure, instead, reaches higher values the larger the connectivity, ap- 
proaching 1 for fc = 208 and large q. In parallel to bump formation, the range of 
q values over which the network retrieves well appears to expand with fc, so that 
for fc = 208 essentially only regular networks fail to retrieve properly. Overall 
the transition from no retrieval to retrieval appears to occur in the same g-region 
as the transition from bumps to no bumps, and for the simulations with full cues 
reported in Fig.|Hlthis g-region shifts leftward, to lower q values, for increasing fc. 

The same lack of a region of coexistence between bump formation and retrieval 
is evident in Fig. [7| reporting simulations with partial cues, p = 50%. Again, 
the bumpiness measure and more prominently the retrieval measure increase 
with more connections per unit, reflecting the larger storage capacity of net- 
works with more connections. In our simulations the number of patterns is 
always fixed at p ~ 5, and the limited size of the network does not allow a 
straightforward application of storage capacity calculations valid for large sys- 
tems |13| . Still, networks with fc = 24 appear to operate above their capacity, 
and also for larger fc the beneficial effect of a more extensive connectivity is in 
any case reflected in higher values for the overlap with the retrieved patterns. 
The major difference with Fig. |H1 is that with partial cues the location of the 
transition between bumps and retrieval does not appear to be much affected by 
fc, hovering between q — 0.45 and q — 0.6. Moreover, for regular and nearly reg- 
ular networks the normalized overlap does approach zero, without any remnant 
magnetization effect; this appears to 'allow' the bumpiness measure to approach 
higher values, suggestive of an active interference between the two phenomena. 
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Bump formation has indeed been analytically shown to lower the storage capac- 
ity of threshold- linear networks [T2| , and it makes sense that the converse be also 
true, that is that even partial retrieval (which implies a proportion of quiescent 
units in the would-be bump) should loosen the bump and weaken its localization. 

These results of both Fig. El and Fig. [7| contradict the expectation that one 
or both of these transitions be simply related to the small-world property of 
the connectivity graph. As shown above, the normalized value of the clustering 
coefhcient, whose decrease marks the right flank of the small-world regime, is 
essentially independent of k, as it is well approximated, in our model, by the 
simple function (1 — q)^. The transition between a localization regime and a 
retrieval regime, instead, to the extent that it can be defined for full cues (with 
the remnant magnetization artefact) clearly depends on fc; while for partial cues 
it occurs at a degree of randomness clearly beyond the small world range. The 
transition does not, therefore, simply reflect a small- world related property. 

4 Discussion 

Although small-world networks have been recently considered in a variety of 
studies, the relevance of a small- world type of connectivity for associative mem- 
ory networks has been touched upon only in a few papers. Bohland and Minai 
|S] compare the retrieval performance of regular, small- world and random net- 
works of an otherwise classic Hopfield model with binary units. They conclude 
that small-world networks with sufficiently large q approach the retrieval per- 
formance of random networks, but with the advantage of a reduced total wire 
length. Note that wire length is measured taking the physical distance between 
connected nodes into account, while path length is just the number of nodes to 
be hopped on in order to connect from one to another node of a pair. This re- 
sult of reduced wire length for similar performance is likely quite general, but it 
relies on the small- world property of the connectivity graph only in the straight- 
forward sense of using up less wiring (which is true also for values of q above 
those of the small- world regime) ; and not in the more non-trivial sense of using 
also the high clustering coefficient of small-world networks. The second result 
obtained by Bohland and Minai is that small-world nets can 'correct' localized 
errors corresponding to incomplete memory patters quicker than more regular 
networks (but slower than more random network) . This again illustrates behav- 
ior intermediate between the extremes of a well performing associative memory 
network with random connections, and of a poorly performing associative mem- 
ory network with regular connections. 

McGraw and Menziger ^Hl similarly compare the final overlap reached by as- 
sociative networks of different connectivity, when started from a full memory 
pattern (this they take to be a measure of pattern stability) and from a very 
corrupted version of a memory pattern (this being a measure of the size of a 
basin of attraction). In terms of both measures small worlds again perform 
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intermediate between regular and random nets, and in fact more similarly to 
random nets for values of g ~ 0.5, intermediate between and 1. They go on to 
discuss the property of another type of connectivity, the scale-free network, in 
which a subset of nodes, the 'hubs', receive more connections than the average, 
and therefore unsurprisingly performs better on measures of associative retrieval 
restricted to the behavior of the hubs themselves. 

In summary, neither of these studies indicates a genuine non-trivial advantage 
of small- world connectivity for associative memories. 

In our study, we considered integrate-and-fire units instead of binary units, 
so as to make more direct contact with cortical networks. More crucially, we 
also investigated, besides associative memory, another behavior, bump forma- 
tion, or localization, which can also be regarded as a sort of emergent network 
computation. Morelli et al in a recent paper ^21 study a seemingly analogous 
phenomenon, i.e. the emergence, in an associative memory with partially or- 
dered connectivity, of discrete domains, in each of which a different memory 
pattern is retrieved. Although broadly consistent with our findings insofar as 
retrieval goes, the phenomenon they study is almost an artefact of the binary 
units used in their network, while we preferred to consider emergent behavior 
relevant to real networks in the brain. We wondered whether connectivity in 
the small- world regime might be reflected in an advantage not in performing as- 
sociative retrieval alone, but in the coexistence of retrieval and bump formation. 

The possibility of such a coexistence is precisely the question that was ad- 
dressed in a recent study that focused on analytically-tractable networks of 
threshold- linear units While the connectivity model was different (it had 
a Gaussian spread with varying width) and unrelated to the notion of small- 
worlds, that study found that retrieval states can indeed be localized, even in 
regular networks, at the price of a relatively minor decrease in storage capacity. 
The very feature that makes threshold-linear units amenable to a full mathe- 
matical analysis, the lack of saturation in their output, may however subtract 
from the generality of the above results. In fact it was noted by Yasser Roudi 
(personal communication) that the units at the center of a retrieval bump need 
to be activated to much higher levels than those at its flanks with similar target 
activation in the memory pattern, or than the very same units when they par- 
ticipate in a non-localized retrieval state. Localized retrieval therefore requires 
the availability of a broad continuous range of activation levels (in the case of 
models with simple input-output units), and it cannot occur with binary units, 
or even with sigmoidal units in which active units all give essentially the same 
output. 

In our present model, we used biologically more relevant integrate-and-fire units, 
whose firing rate, as a function of their inputs, follows, with more complex dy- 
namics, a roughly intermediate behaviour between binary and threshold-linear. 
The absolute refractory period results in a saturation rate (in our case at 
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330Hz), but this ceiling is approached smoothly, and one has to understand 
whether units can use the wide range of firing levels to accomodate retrieval 
specificity within spatially localized states. We failed to observe this coexistence 
in practice, as networks go, with increasing randomness parameter q, from a lo- 
calization regime with no (or very poor) retrieval - to a retrieving regime with 
no bumps. Further, we found that the critical transition q values are beyond 
the boundaries of the small-world region. 

Because of its simulational character, our study cannot be taken to be exhaus- 
tive. For example, the partial retrieval behavior occurring for small q might in 
fact be useful when sufficiently few patterns are stored, p small. In our simula- 
tions, p — 5, but then the whole network was limited in size and connectivity. 
Moreover, a 2D network, which models cortical connectivity better than our 
ID ring, might behave differently. Further, memory patterns defined over only 
parts of the network, which again brings one closer to cortical systems, might 
lead to entirely different conclusions about the coexistence of bump formation 
and associative retrieval. Our simulations, in any case, indicate the impor- 
tance of considering a realistic model of input-output neuronal transform while 
addressing the issue of how to combine autoassociative memory retrieval with 
localization on a cortical map, an issue of crucial importance to understand the 
functions and microcircuitry of the cerebral cortex JS] . 
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Figure 1: (A) Numerical results for the normalized characteristic path length 
L'iq) = [L(q) — Lmin\/[Lmax — Lmin] and the normalized clustering coefficient 
C'{q) = [C{q) — Cmin]/[Cmax — Cmin] SiS 'A fuuctiou of the parameter of random- 
ness (averaged over 10 network realizations, all with the same number k = 41 
of outgoing connections per neuron). Inset: logarithmic x-scale showing the 
small world regime for small q. (B) Comparison of the analytical and numerical 
results for the clustering coefficient C (not normalized, k = il, N = 1000). 




0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 
Parameter of raridomriess 




- Numerical 

- Arialytical 



0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 
Parameter of raridomriess 



19 



Figure 2: Spontaneous emergence of a bump in the activity profile of a regular 
network {q = 0) with an average number of outgoing connections per neuron 
fc = 41. 
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Figure 3: (A) Activity profiles during the simulation for different values of the 
parameter of randomness. Shown is the number of spikes (vertical axis) that 
each neuron (horizontal axis) has fired in a sliding 50 msec time window. (B) 
Overlaps C (vertical axis) of the network activity shown in (A) with each of 
the p = 5 previously stored memory patterns (horizontal axis) - before, during, 
and after a 25%-quality cue for the pattern fi = 2 has been given. 
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Figure 4: Bumpiness of the activity profile (A) and retrieval performance (B) 
as a function of the parameter of randomness for the average of 10 network 
realization, with k ~ Al and p = 1 (error bars are report the stadard deviation 
across realizations). 

A 



O 

£o.6- 




:3 



CO L I I I I L 

0.2 0.4 0.6 0.8 1 

Parameter of randomness 



B 




0.2 0.4 0.6 0.8 1 

Parameter of randomness 



22 



Figure 5: The effect of tfie cue quality p (displayed in the legend) on retrieval 
performance, for the average of 10 network realizations with fc = 41. 
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Figure 6: The effect of changing the number k of connections per neuron on the 
retrieval performance (triangles) and on the bumpiness (dots), for full cues. 



k = 24 k = 58 k = 83 



— ■ — Bumpiness 

1 — • — Retrieval 1 i 




0.5 1 0.5 1 0.5 1 

Parameter of randomness Parameter of randomness Parameter of randomness 

k = 125 k = 165 k = 208 




0.5 1 0.5 1 0.5 1 

Parameter of randomness Parameter of randomness Parameter of randomness 



24 



Figure 7: The effect of changing the number k of connections per neuron on the 
retrieval performance (triangles) and on the bumpiness (dots), for partial cues, 
p = 50%. 




25 



